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Abstract 

Program developers and evaluators need to address the important program accountability question of 
attribution of outcomes. This discussion is a beginning. Starting with some basics - the meaning of 
program, approaches to program theory development, and the nature of atttibution - three types of 
attribution are suggested. An agricultural research verification and extension program provides 
background for the recommendation to use causation attribution in a controlled program environment, and 
associative attribution in a confounded program environment. Implications for evaluation design and 
methodology are discussed. 
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ADDRESSING THE ATTRIBUTION QUESTION IN EXTENSION 

Extension systems in the public sector face increasing accountability demands to justify allocation of 
funds and to demonstrate that effective, need-based programs are in place. External stakeholders want to 
know what difference extension education programs make in the lives of people for whom they are 
intended. In the United States, for example, the federal extension system and a number of state extension 
systems prepare performance-based budgets and report progress/outcomes of their programs against 
predetermined goals. The expectation is that these programs will bring about behavior changes in 
individuals, socio-economic benefits for families, desirable environmental consequences for communities, 
and justifiable returns on tax-supported investments. 

An important question for program developers and evaluators is whether the observed outcome of a 
program is attributable to it. Finding an answer to this question is neither easy nor straightforward 
because (a) non-program variables in the program environment influence outcomes, (b) identifying and 
measuring all relevant program variables may not be feasible or possible, and (c) practical program and 
study considerations mitigate outcomes. Exogenous variables such as non-extension information sources, 
the weather, the agricultural/community infrastructure, and agricultural/consumer market forces, can have 
beneficial or adverse effects on program outcomes. Program variables such as delivery methods, 
contextual factors, and audience (consumer) characteristics also have a bearing on outcomes. While 
certain program variables could be manipulated in the design and implementation of the program, it may 
not be possible to do so for other variables. Furthermore, resource constraints, program design 
requirements and difficulties, and audience considerations are practical implementation and study barriers 
for program developers and evaluators. Answering the attribution question is complex and challenging, 
yet interesting, important, and necessary. 
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The purpose of this paper is to present a rationale and implications of the attribution question, and 
discuss a program evaluation strategy. 

PROGRAM, PROGRAM THEORY, AND ATTRIBUTION 



Program 



Wholey (1987) defines “program” as a set of resources and activities directed toward one or more 
common goals. An administrative interpretation of the U.S. Government Performance and Results Act 
(GPRA, 1993) for the U.S. Cooperative Extension System defines an “extension program” as a series of 
learning experiences designed to bring about desired behavior changes in target clientele (Cooperative 
State Research, Education and Extension Service, 1997). Boyle (1981, p. 14) maintains that “a (major) 
program is not an isolated workshop, event, or activity. . .not a variety of different educational offerings 
available cafeteria style.. .not individualized response to the continuous and urgent requests from 
individuals for information.” These notions suggest that a program should be coherent, organized, and 
focused on accomplishing stated goals. 

Approaches to Program Theory Development in Evaluation 



ERIC 



Chen (1990) posits that most writers in the field of program theory conceptualize it as describing the 
program and how it works. For example, Bickman (1987, p.5): “the construction of a plausible and 
sensible model of how a program is supposed to work.”; Lipsey (1987, p. 7): a set of propositions 
regarding what goes on in the black box during the transformation of input into output; that is, how, via 
treatment inputs, a bad situation is transformed into a better one”; Wholey (1987, p. 78): “program 
resources, activities, and intended program outcomes, and. . .a chain of causal assumptions linking 
program resources, activities, intermediate outcomes, and ultimate goals.” Chen suggests that there is a 
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prescriptive aspect of program theory, namely what the program should accomplish. The prescriptive 
aspect of program theory construction is based on the values of the program developer in that it prescribes 
what actions should be taken, how they should be organized (treatment and implementation), and what 
the outcome criteria should be. 

Patton (1997) conceptualizes three major approaches to theory development. The deductive approach 
uses experimental design methodologies to construct models of the causative relationship between 
program treatments and outcomes. In the inductive approach, the evaluator does field work to generate 
theory, for example, as part of an early evaluability assessment process, or in conjunction with a literature 
review. Patton states, ‘The product of the inductive approach is an empirically derived theoretical model 
of the relationship between program activities and outcomes framed in terms of important contextual 
factors.” (p. 221). The user-focused approach involves working with intended users to extract and specify 
their implicit theory of action about a program. The full chain of events that links inputs to activities, 
activities to immediate outputs, immediate outputs to intermediate outcomes, and intermediate outcomes 
to ultimate goals constitutes a program's theory of action. In contrast to deductive and inductive 
approaches which view programs as replicas of larger phenomena, theories of action are specific to a 

particular program. 

Bennett (1979) conceptualizes a relationship between the “chain of events” in a program and the 
“levels of evidence” needed for evaluation of extension programs . This chain of events enables one to 
construct the program's theory of action: 

a. Inputs (resources) must be assembled to get the program started. 

b. Activities are undertaken with available resources. 

c. Program participants engage in program activities. 

d. Participants react to what they experience. 
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e. As a result of what they experience, changes in knowledge, skills, attitudes, and aspirations occur (if 
the program is effective). 

f. Behavior and practice changes follow knowledge and attitude changes. 

g. Ultimate impacts result, both intended and unintended. 

Attribution 

Is attribution the right question for extension? We start with this rhetorical question because if we 
define attribution solely as causality the question defies a definitive answer. How can we in Extension, 
working in a practical, real-life setting with people, families and communities, satisfy the strict cntenon 
of causation, metaphorically defined by Scriven as the relation between mosquitoes and mosquito bites 
(Scriven, 1991)? It stands to reason that the notion of attribution needs to be broadened beyond strict 

causality. 



In a sense, causality is implicit in a program's theory of action since means are linked to ends in an 
hierarchical relationship. The linkages suggest a logical chain or progression in the program, in that each 
program event is the outcome of the successful attainment of the preceding event and, in turn, is a 
precondition to attainment of the next higher event. In practice, it would appear that the meaning of 
attribution should be moderated to include on an attribution continuum, causality at one end, inference at 
the other, and association in between. 

Relating this continuum to a program's theory of action is instructive. Obviously, cause-effect 
linkages are transparent in the series of links between events. However, the apparent temporal logic of 
immediate to intermediate to ultimate outcomes cannot be rigid in a real life application of the program. 
In practice, the components, links and stages of a program are highly interdependent and dynamically 
interrelated (Patton, 1997). For example, a positive attitude toward learning helps students learn better, 
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but more knowledge can also change negative attitudes. This suggests associative attribution in that there 
is an association or relationship but one is not certain about the direction of the relationship. By 
inferential attribution is meant a value-based, judgmental perception of the program developer/evaluator 
describing what happened in a program, then inferring reasons and providing supporting explanation for 
the program's role in the outcome. 

DETERMINING ATTRIBUTION IN EXTENSION: DESIGN 
AND METHODOLOGICAL CONSIDERATIONS 



Can experimental and quasi-experimental designs be used in evaluating extension work to establish 
causality? To a limited extent, this is possible by (a) using trend data to examine the history of, for 
example, a community's performance before an initiative has begun and after it has been implemented, 
and (b) using contrived situations in which one or more variables of interest are controlled in conventional 
experimental designs such as time series and comparison groups. Some obvious reasons why extension 
programs targeted to individuals, families or communities do not lend themselves easily to experimental 
design are the smallness of change, time duration required to achieve impact, number of cases available to 
study, high cost, and the ethics of withholding education from a specific population. 

It would appear that associative attribution and inferential attribution are more suited to extension 
program evaluation. General evaluation techniques in these cases include. 

a. Combining multiple sources of evidence, including primary and secondary data sources, and groups 
that influence program performance and outcome so as to achieve a confluence of results. 

b. Using multiple tools such as qualitative and quantitative methods, and suitable analytical techniques 

to strengthen attribution. 
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c. Using the theory of action approach to show how a program's logic falls into place and whether the 
program was “on track” with this logic. 

d. Systematically examining factors other than the program treatment that might influence outcomes. 

e. Documenting early outcomes that lead to longer-term results to show how means leads to desired 
ends. This also helps to improve the program as it is being conducted. 

ELUCIDATING THE ATTRIBUTION QUESTION: AN EXAMPLE 

An example of an ongoing extension program of a state extension service in the United States 
illustrates some considerations in addressing the attribution question. In general, the following steps are 
visualized in planning the evaluation of a program focused on this question. 

a. Involving relevant stakeholders in developing the program's theory of action because they are the 
intended users of results. Bringing them in at this stage (a) gets legitimation and some guarantee that 
evaluation findings will be used, (b) clarifies the attribution question, i.e., causal, associative or 
inferential, (c) provides stakeholder expectations in this regard, and (d) guides the program 
developer/evaluator to frame one or more evaluation questions, and consider design and 
methodology. 

b. Deciding level(s) of impact to evaluate and the type(s) of attribution to be determined. This decision 
will suggest evaluation design and methodology — instrumentation, data collection, statistical 
techniques. 
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Example: Soybean Research Verification and Extension Program 

Background . Soybean is a major commodity in the state, contributing $252 million to the annual 
economy. In 1996, there were 5,697 producers, who produced 35.9 million bushels on 1 million acres. 

The state average yield is about 26 bushels/acre. 

Commodity research and extension programs over the years have shown that yield is increased when 
farmers adopt approved practices, tested in research settings and under field conditions, and disseminated 
through an information system. An intensive, holistic approach was introduced in the soybean extension 
program in 1994. A set of cultural practices — improved varieties, tillage (seedbed preparation), planting, 
fertilizing, pest management, irrigation — recommended by soybean researchers was verified under field 
conditions. Farmers who agreed to stay in the program for two growing seasons and follow the 
recommended technology and management system were chosen by county agents as program 
cooperators. Under close technical supervision of county agents and subject-matter specialists, 
cooperating farmers learned how to manage their production operation in a systematic, attentive manner 
by following and adopting the set of cultural practices recommended for their individual operations. Over 
a four-year period, data on yield, cost, and the relative influence on yield and cost of each practice were 
gathered and analyzed on 38 verification fields (one field per farmer per county). Average soybean yield 
from these fields was recorded at 11 bushels per acre more than the state average, and the combined fixed 
and variable production cost per bushel was $3.78, well below the current price of $6.40 per bushel. 

County agents and specialists expected cooperating farmers to extend the technology and financial 
management system from the small verification fields to their whole farm. Field days and producer 
meetings using cooperating farmers as spokespersons to share the results they achieved were organized in 
the counties as a means of encouraging other farmers. 
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The Program's Theory of Action and Matching Levels of Evidence . How can we evaluate the 
outcome of this program to answer the attribution question? We need first to construct the program's 
theory of action. Figure 1 shows the theory of action and matching levels of evidence for the program. 

The program theorizes that resources such as faculty time, technology/management systems guidance, 
and education materials are provided by the state extension service through weekly visits to cooperators 
during the crop's growing season, and organized production meetings and field days using cooperator 
fields as demonstrations for other farmers. Cooperators on their part provide production inputs, and 
incorporate technical advice of extension faculty into their operations. They develop a positive regard for 
these interactions, learn the value of attention to detail and timely management of the production system, 
as well as any new technology, and share the results of their experience with other farmers in field days, 
production meetings, and informal contacts. Cooperators practice the recommended technology system 
on a small scale in field verification plots and eventually extend the system to the entire farm operation. 

As a result, soybean yields and, potentially, incomes are increased on the farms of cooperators. Other 
farmers begin to evaluate the technology system on a small scale and also experience increases in yield 
and income. 

Evidence for levels 1-4 (inputs, activities, farmer participation, and reactions) can be gathered from 
program records and attributed with a high degree of certainty to the extension program because there is a 
close association among extension faculty, program activities, and cooperators. Levels 5-7 indicate the 
outcomes of the program. In general, this evidence is more difficult to plan for and gather, and requires 
greater resources. The task becomes more complex, time-consuming, and difficult when we want to 
attribute observed effects or outcomes to the program and its activities. Furthermore, program effects that 
are not planned for or seen by the evaluator, may not have had sufficient time to appear, or may be hard to 
measure, complicate the task. This suggests that the total outcome of a program is likely to be 




underestimated. 
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It is generally agreed that Extension is a major player in the information dissemination system since it 
is a prime source from which research information in the public sector originates. But it is not the only 
source. Increasingly, the Natural Resources Conservation Service is a major source with advice on 
irrigation, soil conservation, etc. Also, Experiment Stations and the Agricultural Research Service are 
public sector sources. Other players - agricultural consultants, dealers, the print and broadcast media, and 
friends and neighbors - have a significant role in providing information to farmers. Adoption-diffusion 
research in agriculture has shown that a variety of personal, social, economic, and technological factors 
are related to knowledge, skills and attitude changes, and adoption of technological innovations. The 
inference that economic and social benefits occur as a result of behavior and practice changes envisioned 
in a program's theory of action is confounded by environmental factors. In this kind of dynamic 
environment, therefore, program evaluation has to be focused on a key question, and the evaluation 
design has to be flexible and include variables which have an influence on or help provide answers to the 
question. 

In our example, the evaluation question could be focused on (a) determining the effects on farmers 
cooperating in the research verification program, or (b) other farmers participating in the broader 
extension program. In either case, one could determine impact at Level 5, 6 or 7. 

In the first question, since cooperating farmers constitute a homogeneous group with whom extension 
faculty have worked intensively over two growing seasons, Level 7 effect can be examined. Records of 
cultural practices recommended by extension faculty and followed by farmers are available. Yield and 
cost data are also available. Income received by each farmer, and return to cost can be calculated for each 
field plot. Additional income can be determined by extrapolating field plot results to the total operation. 

It would appear that the economic benefit to cooperating farmers is totally, or partially, attributable to the 
research verification and extension program. The basis for this assertion is that the program is highly 
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structured through the chain of events identified in the program's theory of action. Verifying linkages 
between successive events will strengthen this assertion, and test the theory. 

The focus of the second question is farmers who have not participated in the research verification 
program but are involved in the extension phase. They are a broader and more challenging group for 
studying attribution. Their contact with extension faculty is limited to field days and production 
meetings, informal contacts with cooperators, and routine extension education activities and materials. 
Learning and adopting recommended practices to ultimately impact production and income will depend 
on a number of factors in their personal situations, the farming and education context, and successful 
experiences. Evaluating outcomes in this typical extension scenario will require that program and non- 
program (exogenous) variables be identified, and appropriate statistical techniques applied. In this 
situation, the evaluator would probably focus on determining associative or inferential attribution. 

SUMMARY AND CONCLUSIONS 

We began this discussion with the question - “How do we determine attribution of outcome to a 
program?” Placing the question in the context of a program evaluation strategy, we stated that it was 
necessary for program developers and evaluators to: 

a) understand that a program is a coherent, organized set of resources and activities focused on one or 
more common goals and not isolated, disconnected events; 

b) construct prior to implementation of a program, the program’s theory of action specifying the “chain 
of program events” and the “kinds of evidence” appropriate for each event in the chain; 

c) decide which program event or events to evaluate, the type(s) of attribution to be determined, and the 
appropriate evaluation design and methodology to use. 
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Discussions of the question in the above light, and the example of an agricultural research and 
extension program which was given to illustrate this approach, lead to several propositions. 

a. Attributing outcome to a program with a high level of certainty is confounded by non-program 
influences and those program variables which cannot be or are not considered by the evaluation. As 
such, causation attribution in extension programming is likely to be rare and restricted to programs 
and program environments that are homogeneous and tightly controlled, with one or a small number 
of treatment variables. 

b. A majority of extension programs lend themselves to two other types of attribution - associative and 
inferential. In these cases, the evaluator cannot be certain that a particular program results in some 
definite outcome, but the connection between program and outcome could be described using suitable 
techniques such as multiple sources of evidence, multiple tools, analytical techniques, and anecdotal 
material. 

c. A caveat that undergirds program evaluation efforts and the attribution issue should be pointed out. 
Ostensibly, program effects that are not planned for or discerned by the evaluator, that may not have 
sufficient time to become transparent, or that may be hard, even impossible, to measure complicates 
the task of evaluation. It would appear, therefore, that the total effect of a program is likely to be 
invariably underestimated. 

These propositions and the accompanying discussion are offered to program developers and 
evaluators as food for thought and empirical research as we try to ourselves understand and promote 
understanding of this issue among educators and other stakeholders in the extension system. 
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FIGURE 1. SOYBEAN RESEARCH VERIFICATION AND EXTENSION PROGRAM 
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